【1】MARS: An Instance-aware, Modular and Realistic Simulator for Autonomous Driving

【URL】https://arxiv.org/abs/2311.01017v1

【Time】2023-07-27

一、研究领域

自动驾驶，NeRF，Simulator

二、研究动机

realistic sensor simulation可以通过模拟解决真实情况下难以收集到的corner case。作者希望提出一个用于构建组合神经辐射场的模块化框架，可以在其中对户外驾驶场景进行真实的传感器模拟。

三、方法与技术

对每个前景实例和背景结点进行组合建模：

（1）分别查询背景和前景节点，，每个节点采样一组3D points，通过特定的神经网络获取相应的point properties（RGB，density，semantics等）

（2）来自背景和前景节点的所有光线样本被组合并进行体积渲染，以产生逐像素渲染结果

Fig. 1. Pipeline. Left: We first calculate the ray-box intersection of the queried ray r and all visible instance bounding boxes {Bij}. For the background node, we directly use the selected scene representation model and the chosen sampler to infer pointwise properties, as in conventional NeRFs. For the foreground nodes, the ray is first transformed into the instance frame as ro before being processed through foreground node representations (Sec. 2.1). Right: All the samples are composed and rendered into RGB images, depth maps, and semantics (Sec. 2.2).

Fig. 1. Pipeline. Left: We first calculate the ray-box intersection of the queried ray r and all visible instance bounding boxes {Bij}. For the background node, we directly use the selected scene representation model and the chosen sampler to infer pointwise properties, as in conventional NeRFs. For the foreground nodes, the ray is first transformed into the instance frame as ro before being processed through foreground node representations (Sec. 2.1). Right: All the samples are composed and rendered into RGB images, depth maps, and semantics (Sec. 2.2).

四、总结

Pipeline上对各个节点包括天空都有非常细致的建模，这就很…很NeRF。框架是开源的，试了一下，虽然慢且环境不好配（都怪tinycudann哼哼），但是确实好用。